Cross-modal exposure restores multisensory enhancement after hemianopia.

Hemianopia is a common consequence of unilateral damage to visual cortex that manifests as a profound blindness in contralesional space. A noninvasive cross-modal (visual-auditory) exposure paradigm has been developed in an animal model to ameliorate this disorder. Repeated stimulation of a visual-auditory stimulus restores overt responses to visual stimuli in the blinded hemifield. It is believed to accomplish this by enhancing the visual sensitivity of circuits remaining after a lesion of visual cortex; in particular, circuits involving the multisensory neurons of the superior colliculus. Neurons in this midbrain structure are known to integrate spatiotemporally congruent visual and auditory signals to amplify their responses, which, in turn, enhances behavioral performance. Here we evaluated the relationship between the rehabilitation of hemianopia and this process of multisensory integration. Induction of hemianopia also eliminated multisensory enhancement in the blinded hemifield. Both vision and multisensory enhancement rapidly recovered with the rehabilitative cross-modal exposures. However, although both reached pre-lesion levels at similar rates, they did so with different spatial patterns. The results suggest that the capability for multisensory integration and enhancement is not a pre-requisite for visual recovery in hemianopia, and that the underlying mechanisms for recovery may be more complex than currently appreciated.

Predictability alters multisensory responses by modulating unisensory inputs.

The multisensory (deep) layers of the superior colliculus (SC) play an important role in detecting, localizing, and guiding orientation responses to salient events in the environment. Essential to this role is the ability of SC neurons to enhance their responses to events detected by more than one sensory modality and to become desensitized ('attenuated' or 'habituated') or sensitized ('potentiated') to events that are predictable via modulatory dynamics. To identify the nature of these modulatory dynamics, we examined how the repetition of different sensory stimuli affected the unisensory and multisensory responses of neurons in the cat SC. Neurons were presented with 2HZ stimulus trains of three identical visual, auditory, or combined visual-auditory stimuli, followed by a fourth stimulus that was either the same or different ('switch'). Modulatory dynamics proved to be sensory-specific: they did not transfer when the stimulus switched to another modality. However, they did transfer when switching from the visual-auditory stimulus train to either of its modality-specific component stimuli and vice versa. These observations suggest that predictions, in the form of modulatory dynamics induced by stimulus repetition, are independently sourced from and applied to the modality-specific inputs to the multisensory neuron. This falsifies several plausible mechanisms for these modulatory dynamics: they neither produce general changes in the neuron's transform, nor are they dependent on the neuron's output.

Ameliorating Hemianopia with Multisensory Training.

Hemianopia (unilateral blindness), a common consequence of stroke and trauma to visual cortex, is a debilitating disorder for which there are few treatments. Research in an animal model has suggested that visual-auditory stimulation therapy, which exploits the multisensory architecture of the brain, may be effective in restoring visual sensitivity in hemianopia. It was tested in two male human patients who were hemianopic for at least 8 months following a stroke. The patients were repeatedly exposed to congruent visual-auditory stimuli within their blinded hemifield during 2 h sessions over several weeks. The results were dramatic. Both recovered the ability to detect and describe visual stimuli throughout their formerly blind field within a few weeks. They could also localize these stimuli, identify some of their features, and perceive multiple visual stimuli simultaneously in both fields. These results indicate that the multisensory therapy is a rapid and effective method for restoring visual function in hemianopia.SIGNIFICANCE STATEMENT Hemianopia (blindness on one side of space) is widely considered to be a permanent disorder. Here, we show that a simple multisensory training paradigm can ameliorate this disorder in human patients.

Noise-rearing precludes the behavioral benefits of multisensory integration.

Concordant visual-auditory stimuli enhance the responses of individual superior colliculus (SC) neurons. This neuronal capacity for "multisensory integration" is not innate: it is acquired only after substantial cross-modal (e.g. auditory-visual) experience. Masking transient auditory cues by raising animals in omnidirectional sound ("noise-rearing") precludes their ability to obtain this experience and the ability of the SC to construct a normal multisensory (auditory-visual) transform. SC responses to combinations of concordant visual-auditory stimuli are depressed, rather than enhanced. The present experiments examined the behavioral consequence of this rearing condition in a simple detection/localization task. In the first experiment, the auditory component of the concordant cross-modal pair was novel, and only the visual stimulus was a target. In the second experiment, both component stimuli were targets. Noise-reared animals failed to show multisensory performance benefits in either experiment. These results reveal a close parallel between behavior and single neuron physiology in the multisensory deficits that are induced when noise disrupts early visual-auditory experience.

Association Cortex Is Essential to Reverse Hemianopia by Multisensory Training.

Hemianopia induced by unilateral visual cortex lesions can be resolved by repeatedly exposing the blinded hemifield to auditory-visual stimuli. This rehabilitative "training" paradigm depends on mechanisms of multisensory plasticity that restore the lost visual responsiveness of multisensory neurons in the ipsilesional superior colliculus (SC) so that they can once again support vision in the blinded hemifield. These changes are thought to operate via the convergent visual and auditory signals relayed to the SC from association cortex (the anterior ectosylvian sulcus [AES], in cat). The present study tested this assumption by cryogenically deactivating ipsilesional AES in hemianopic, anesthetized cats during weekly multisensory training sessions. No signs of visual recovery were evident in this condition, even after providing animals with up to twice the number of training sessions required for effective rehabilitation. Subsequent training under the same conditions, but with AES active, reversed the hemianopia within the normal timeframe. These results indicate that the corticotectal circuit that is normally engaged in SC multisensory plasticity has to be operational for the brain to use visual-auditory experience to resolve hemianopia.

Multisensory enhancement of overt behavior requires multisensory experience.

The superior colliculus (SC) is richly endowed with neurons that integrate cues from different senses to enhance their physiological responses and the overt behaviors they mediate. However, in the absence of experience with cross-modal combinations (e.g., visual-auditory), they fail to develop this characteristic multisensory capability: Their multisensory responses are no greater than their most effective unisensory responses. Presumably, this impairment in neural development would be reflected as corresponding impairments in SC-mediated behavioral capabilities such as detection and localization performance. Here, we tested that assumption directly in cats raised to adulthood in darkness. They, along with a normally reared cohort, were trained to approach brief visual or auditory stimuli. The animals were then tested with these stimuli individually and in combination under ambient light conditions consistent with their rearing conditions and home environment as well as under the opposite lighting condition. As expected, normally reared animals detected and localized the cross-modal combinations significantly better than their individual component stimuli. However, dark-reared animals showed significant defects in multisensory detection and localization performance. The results indicate that a physiological impairment in single multisensory SC neurons is predictive of an impairment in overt multisensory behaviors.

Stimulus value gates multisensory integration.

The brain enhances its perceptual and behavioral decisions by integrating information from its multiple senses in what are believed to be optimal ways. This phenomenon of "multisensory integration" appears to be pre-conscious, effortless, and highly efficient. The present experiments examined whether experience could modify this seemingly automatic process. Cats were trained in a localization task in which congruent pairs of auditory-visual stimuli are normally integrated to enhance detection and orientation/approach performance. Consistent with the results of previous studies, animals more reliably detected and approached cross-modal pairs than their modality-specific component stimuli, regardless of whether the pairings were novel or familiar. However, when provided evidence that one of the modality-specific component stimuli had no value (it was not rewarded) animals ceased integrating it with other cues, and it lost its previous ability to enhance approach behaviors. Cross-modal pairings involving that stimulus failed to elicit enhanced responses even when the paired stimuli were congruent and mutually informative. However, the stimulus regained its ability to enhance responses when it was associated with reward. This suggests that experience can selectively block access of stimuli (i.e., filter inputs) to the multisensory computation. Because this filtering process results in the loss of useful information, its operation and behavioral consequences are not optimal. Nevertheless, the process can be of substantial value in natural environments, rich in dynamic stimuli, by using experience to minimize the impact of stimuli unlikely to be of biological significance, and reducing the complexity of the problem of matching signals across the senses.

Experience Creates the Multisensory Transform in the Superior Colliculus.

Although the ability to integrate information across the senses is compromised in some individuals for unknown reasons, similar defects have been observed when animals are reared without multisensory experience. The experience-dependent development of multisensory integration has been studied most extensively using the visual-auditory neuron of the cat superior colliculus (SC) as a neural model. In the normally-developed adult, SC neurons react to concordant visual-auditory stimuli by integrating their inputs in real-time to produce non-linearly amplified multisensory responses. However, when prevented from gathering visual-auditory experience, their multisensory responses are no more robust than their responses to the individual component stimuli. The mechanisms operating in this defective state are poorly understood. Here we examined the responses of SC neurons in "naïve" (i.e., dark-reared) and "neurotypic" (i.e., normally-reared) animals on a millisecond-by-millisecond basis to determine whether multisensory experience changes the operation by which unisensory signals are converted into multisensory outputs (the "multisensory transform"), or whether it changes the dynamics of the unisensory inputs to that transform (e.g., their synchronization and/or alignment). The results reveal that the major impact of experience was on the multisensory transform itself. Whereas neurotypic multisensory responses exhibited non-linear amplification near their onset followed by linear amplification thereafter, the naive responses showed no integration in the initial phase of the response and a computation consistent with competition in its later phases. The results suggest that multisensory experience creates an entirely new computation by which convergent unisensory inputs are used cooperatively to enhance the physiological salience of cross-modal events and thereby facilitate normal perception and behavior.

Using superior colliculus principles of multisensory integration to reverse hemianopia.

The diversity of our senses conveys many advantages; it enables them to compensate for one another when needed, and the information they provide about a common event can be integrated to facilitate its processing and, ultimately, adaptive responses. These cooperative interactions are produced by multisensory neurons. A well-studied model in this context is the multisensory neuron in the output layers of the superior colliculus (SC). These neurons integrate and amplify their cross-modal (e.g., visual-auditory) inputs, thereby enhancing the physiological salience of the initiating event and the probability that it will elicit SC-mediated detection, localization, and orientation behavior. Repeated experience with the same visual-auditory stimulus can also increase the neuron's sensitivity to these individual inputs. This observation raised the possibility that such plasticity could be engaged to restore visual responsiveness when compromised. For example, unilateral lesions of visual cortex compromise the visual responsiveness of neurons in the multisensory output layers of the ipsilesional SC and produces profound contralesional blindness (hemianopia). The possibility that multisensory plasticity could restore the visual responses of these neurons, and reverse blindness, was tested in the cat model of hemianopia. Hemianopic subjects were repeatedly presented with spatiotemporally congruent visual-auditory stimulus pairs in the blinded hemifield on a daily or weekly basis. After several weeks of this multisensory exposure paradigm, visual responsiveness was restored in SC neurons and behavioral responses were elicited by visual stimuli in the previously blind hemifield. The constraints on the effectiveness of this procedure proved to be the same as those constraining SC multisensory plasticity: whereas repetitions of a congruent visual-auditory stimulus was highly effective, neither exposure to its individual component stimuli, nor to these stimuli in non-congruent configurations was effective. The restored visual responsiveness proved to be robust, highly competitive with that in the intact hemifield, and sufficient to support visual discrimination.

Reversing Hemianopia by Multisensory Training Under Anesthesia.

Hemianopia is characterized by blindness in one half of the visual field and is a common consequence of stroke and unilateral injury to the visual cortex. There are few effective rehabilitative strategies that can relieve it. Using the cat as an animal model of hemianopia, we found that blindness induced by lesions targeting all contiguous areas of the visual cortex could be rapidly reversed by a non-invasive, multisensory (auditory-visual) exposure procedure even while animals were anesthetized. Surprisingly few trials were required to reinstate vision in the previously blind hemisphere. That rehabilitation was possible under anesthesia indicates that the visuomotor behaviors commonly believed to be essential are not required for this recovery, nor are factors such as attention, motivation, reward, or the various other cognitive features that are generally thought to facilitate neuro-rehabilitative therapies.

Multisensory Integration and the Society for Neuroscience: Then and Now.

The operation of our multiple and distinct sensory systems has long captured the interest of researchers from multiple disciplines. When the Society was founded 50 years ago to bring neuroscience research under a common banner, sensory research was largely divided along modality-specific lines. At the time, there were only a few physiological and anatomical observations of the multisensory interactions that powerfully influence our everyday perception. Since then, the neuroscientific study of multisensory integration has increased exponentially in both volume and diversity. From initial studies identifying the overlapping receptive fields of multisensory neurons, to subsequent studies of the spatial and temporal principles that govern the integration of multiple sensory cues, our understanding of this phenomenon at the single-neuron level has expanded to include a variety of dimensions. We now can appreciate how multisensory integration can alter patterns of neural activity in time, and even coordinate activity among populations of neurons across different brain areas. There is now a growing battery of sophisticated empirical and computational techniques that are being used to study this process in a number of models. These advancements have not only enhanced our understanding of this remarkable process in the normal adult brain, but also its underlying circuitry, requirements for development, susceptibility to malfunction, and how its principles may be used to mitigate malfunction.

Using the Principles of Multisensory Integration to Reverse Hemianopia.

Hemianopia can be rehabilitated by an auditory-visual "training" procedure, which restores visual responsiveness in midbrain neurons indirectly compromised by the cortical lesion and reinstates vision in contralesional space. Presumably, these rehabilitative changes are induced via mechanisms of multisensory integration/plasticity. If so, the paradigm should fail if the stimulus configurations violate the spatiotemporal principles that govern these midbrain processes. To test this possibility, hemianopic cats were provided spatially or temporally noncongruent auditory-visual training. Rehabilitation failed in all cases even after approximately twice the number of training trials normally required for recovery, and even after animals learned to approach the location of the undetected visual stimulus. When training was repeated with these stimuli in spatiotemporal concordance, hemianopia was resolved. The results identify the conditions needed to engage changes in remaining neural circuits required to support vision in the absence of visual cortex, and have implications for rehabilitative strategies in human patients.

Interhemispheric visual competition after multisensory reversal of hemianopia.

Unilateral lesions of visual cortex have the secondary consequence of suppressing visual circuits in the midbrain superior colliculus (SC), collectively producing blindness in contralesional space ("hemianopia"). Recent studies have demonstrated that SC visual responses and contralesional vision can be reinstated by a non-invasive multisensory training procedure in which spatiotemporally concordant visual-auditory pairs are repeatedly presented within the blind hemifield. Despite this recovery of visual responsiveness, the loss of visual cortex was expected to result in permanent deficits in that hemifield, especially when visual events in both hemifields compete for attention and access to the brain's visuomotor circuitry. This was evaluated in the present study in a visual choice paradigm in which the two visual hemifields of recovered cats were simultaneously stimulated with equally valent visual targets. Surprisingly, the expected disparity was not found, and some animals even preferred stimuli presented in the previously blind hemifield. This preference persisted across multiple stimulus intensity levels and there was no indication that animals were less aware of cues in the previously blind hemifield than in its spared counterpart. Furthermore, when auditory cues were combined with visual cues, the enhanced performance they produced on a visual task was no greater in the normal than in the previously blind hemifield. These observations suggest that the multisensory rehabilitation paradigm revealed greater inherent visual information processing potential in the previously blind hemifield than was believed possible given the loss of visual cortex.

Cross-Modal Competition: The Default Computation for Multisensory Processing.

Mature multisensory superior colliculus (SC) neurons integrate information across the senses to enhance their responses to spatiotemporally congruent cross-modal stimuli. The development of this neurotypic feature of SC neurons requires experience with cross-modal cues. In the absence of such experience the response of an SC neuron to congruent cross-modal cues is no more robust than its response to the most effective component cue. This "default" or "naive" state is believed to be one in which cross-modal signals do not interact. The present results challenge this characterization by identifying interactions between visual-auditory signals in male and female cats reared without visual-auditory experience. By manipulating the relative effectiveness of the visual and auditory cross-modal cues that were presented to each of these naive neurons, an active competition between cross-modal signals was revealed. Although contrary to current expectations, this result is explained by a neuro-computational model in which the default interaction is mutual inhibition. These findings suggest that multisensory neurons at all maturational stages are capable of some form of multisensory integration, and use experience with cross-modal stimuli to transition from their initial state of competition to their mature state of cooperation. By doing so, they develop the ability to enhance the physiological salience of cross-modal events thereby increasing their impact on the sensorimotor circuitry of the SC, and the likelihood that biologically significant events will elicit SC-mediated overt behaviors.SIGNIFICANCE STATEMENT The present results demonstrate that the default mode of multisensory processing in the superior colliculus is competition, not non-integration as previously characterized. A neuro-computational model explains how these competitive dynamics can be implemented via mutual inhibition, and how this default mode is superseded by the emergence of cooperative interactions during development.

Development of the Mechanisms Governing Midbrain Multisensory Integration.

The ability to integrate information across multiple senses enhances the brain's ability to detect, localize, and identify external events. This process has been well documented in single neurons in the superior colliculus (SC), which synthesize concordant combinations of visual, auditory, and/or somatosensory signals to enhance the vigor of their responses. This increases the physiological salience of crossmodal events and, in turn, the speed and accuracy of SC-mediated behavioral responses to them. However, this capability is not an innate feature of the circuit and only develops postnatally after the animal acquires sufficient experience with covariant crossmodal events to form links between their modality-specific components. Of critical importance in this process are tectopetal influences from association cortex. Recent findings suggest that, despite its intuitive appeal, a simple generic associative rule cannot explain how this circuit develops its ability to integrate those crossmodal inputs to produce enhanced multisensory responses. The present neurocomputational model explains how this development can be understood as a transition from a default state in which crossmodal SC inputs interact competitively to one in which they interact cooperatively. Crucial to this transition is the operation of a learning rule requiring coactivation among tectopetal afferents for engagement. The model successfully replicates findings of multisensory development in normal cats and cats of either sex reared with special experience. In doing so, it explains how the cortico-SC projections can use crossmodal experience to craft the multisensory integration capabilities of the SC and adapt them to the environment in which they will be used.SIGNIFICANCE STATEMENT The brain's remarkable ability to integrate information across the senses is not present at birth, but typically develops in early life as experience with crossmodal cues is acquired. Recent empirical findings suggest that the mechanisms supporting this development must be more complex than previously believed. The present work integrates these data with what is already known about the underlying circuit in the midbrain to create and test a mechanistic model of multisensory development. This model represents a novel and comprehensive framework that explains how midbrain circuits acquire multisensory experience and reveals how disruptions in this neurotypic developmental trajectory yield divergent outcomes that will affect the multisensory processing capabilities of the mature brain.

The normal environment delays the development of multisensory integration.

Multisensory neurons in animals whose cross-modal experiences are compromised during early life fail to develop the ability to integrate information across those senses. Consequently, they lack the ability to increase the physiological salience of the events that provide the convergent cross-modal inputs. The present study demonstrates that superior colliculus (SC) neurons in animals whose visual-auditory experience is compromised early in life by noise-rearing can develop visual-auditory multisensory integration capabilities rapidly when periodically exposed to a single set of visual-auditory stimuli in a controlled laboratory paradigm. However, they remain compromised if their experiences are limited to a normal housing environment. These observations seem counterintuitive given that multisensory integrative capabilities ordinarily develop during early life in normal environments, in which a wide variety of sensory stimuli facilitate the functional organization of complex neural circuits at multiple levels of the neuraxis. However, the very richness and inherent variability of sensory stimuli in normal environments will lead to a less regular coupling of any given set of cross-modal cues than does the otherwise "impoverished" laboratory exposure paradigm. That this poses no significant problem for the neonate, but does for the adult, indicates a maturational shift in the requirements for the development of multisensory integration capabilities.

Multisensory Integration Uses a Real-Time Unisensory-Multisensory Transform.

The manner in which the brain integrates different sensory inputs to facilitate perception and behavior has been the subject of numerous speculations. By examining multisensory neurons in cat superior colliculus, the present study demonstrated that two operational principles are sufficient to understand how this remarkable result is achieved: (1) unisensory signals are integrated continuously and in real time as soon as they arrive at their common target neuron and (2) the resultant multisensory computation is modified in shape and timing by a delayed, calibrating inhibition. These principles were tested for descriptive sufficiency by embedding them in a neurocomputational model and using it to predict a neuron's moment-by-moment multisensory response given only knowledge of its responses to the individual modality-specific component cues. The predictions proved to be highly accurate, reliable, and unbiased and were, in most cases, not statistically distinguishable from the neuron's actual instantaneous multisensory response at any phase throughout its entire duration. The model was also able to explain why different multisensory products are often observed in different neurons at different time points, as well as the higher-order properties of multisensory integration, such as the dependency of multisensory products on the temporal alignment of crossmodal cues. These observations not only reveal this fundamental integrative operation, but also identify quantitatively the multisensory transform used by each neuron. As a result, they provide a means of comparing the integrative profiles among neurons and evaluating how they are affected by changes in intrinsic or extrinsic factors.SIGNIFICANCE STATEMENT Multisensory integration is the process by which the brain combines information from multiple sensory sources (e.g., vision and audition) to maximize an organism's ability to identify and respond to environmental stimuli. The actual transformative process by which the neural products of multisensory integration are achieved is poorly understood. By focusing on the millisecond-by-millisecond differences between a neuron's unisensory component responses and its integrated multisensory response, it was found that this multisensory transform can be described by two basic principles: unisensory information is integrated in real time and the multisensory response is shaped by calibrating inhibition. It is now possible to use these principles to predict a neuron's multisensory response accurately armed only with knowledge of its unisensory responses.

Pulsed Stimuli Elicit More Robust Multisensory Enhancement than Expected.

Neurons in the superior colliculus (SC) integrate cross-modal inputs to generate responses that are more robust than to either input alone, and are frequently greater than their sum (superadditive enhancement). Previously, the principles of a real-time multisensory transform were identified and used to accurately predict a neuron's responses to combinations of brief flashes and noise bursts. However, environmental stimuli frequently have more complex temporal structures that elicit very different response dynamics than previously examined. The present study tested whether such stimuli (i.e., pulsed) would be treated similarly by the multisensory transform. Pulsing visual and auditory stimuli elicited responses composed of higher discharge rates that had multiple peaks temporally aligned to the stimulus pulses. Combinations pulsed cues elicited multiple peaks of superadditive enhancement within the response window. Measured over the entire response, this resulted in larger enhancements than expected given enhancements elicited by non-pulsed ("sustained") stimuli. However, as with sustained stimuli, the dynamics of multisensory responses to pulsed stimuli were highly related to the temporal dynamics of the unisensory inputs. This suggests that the specific characteristics of the multisensory transform are not determined by the external features of the cross-modal stimulus configuration; rather the temporal structure and alignment of the unisensory inputs is the dominant driving factor in the magnitudes of the multisensory product.

Multisensory Plasticity in Superior Colliculus Neurons is Mediated by Association Cortex.

The ability to integrate information from different senses, and thereby facilitate detecting and localizing events, normally develops gradually in cat superior colliculus (SC) neurons as experience with cross-modal events is acquired. Here, we demonstrate that the portal for this experience-based change is association cortex. Unilaterally deactivating this cortex whenever visual-auditory events were present resulted in the failure of ipsilateral SC neurons to develop the ability to integrate those cross-modal inputs, even though they retained the ability to respond to them. In contrast, their counterparts in the opposite SC developed this capacity normally. The deficits were eliminated by providing cross-modal experience when cortex was active. These observations underscore the collaborative developmental processes that take place among different levels of the neuraxis to adapt the brain's multisensory (and sensorimotor) circuits to the environment in which they will be used.

Multisensory training reverses midbrain lesion-induced changes and ameliorates haemianopia.

Failure to attend to visual cues is a common consequence of visual cortex injury. Here, we report on a behavioural strategy whereby cross-modal (auditory-visual) training reinstates visuomotor competencies in animals rendered haemianopic by complete unilateral visual cortex ablation. The re-emergence of visual behaviours is correlated with the reinstatement of visual responsiveness in deep layer neurons of the ipsilesional superior colliculus (SC). This functional recovery is produced by training-induced alterations in descending influences from association cortex that allowed these midbrain neurons to once again transform visual cues into appropriate orientation behaviours. The findings underscore the inherent plasticity and functional breadth of phylogenetically older visuomotor circuits that can express visual capabilities thought to have been subsumed by more recently evolved brain regions. These observations suggest the need for reevaluating current concepts of functional segregation in the visual system and have important implications for strategies aimed at ameliorating trauma-induced visual deficits in humans.